Add `ScalarValue::RunEndEncoded` variant by Jefffrey · Pull Request #19895 · apache/datafusion

Jefffrey · 2026-01-20T08:28:40Z

Which issue does this PR close?

Closes Missing ScalarValue variant for RunEndEncoded #18563

Rationale for this change

Support RunEndEncoded scalar values, similar to how we support for Dictionary.

What changes are included in this PR?

Add new ScalarValue::RunEndEncoded enum variant
Fix ScalarValue::new_default to support Decimal32 and Decimal64
Support RunEndEncoded type in proto for both ScalarValue message and ArrowType message

Are these changes tested?

Added tests.

Are there any user-facing changes?

New variant for ScalarValue

Protobuf changes to support RunEndEncoded type

… type

Jefffrey · 2026-01-24T08:08:37Z

+    /// (run-ends field, value field, value)
+    RunEndEncoded(FieldRef, FieldRef, Box<ScalarValue>),


Mimicking the arrow type where it stores fields:

https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#variant.RunEndEncoded

I tried initially only storing the index DataType and the ScalarValue value, but figured it would be better to try be as accurate as possible 🤔

Jefffrey · 2026-01-24T08:09:05Z

+            | DataType::Decimal32(_, _)
+            | DataType::Decimal64(_, _)


Little fix since we were missing these

Jefffrey · 2026-01-24T08:09:24Z


-            // Unsupported types for now
-            _ => {
+            DataType::ListView(_) | DataType::LargeListView(_) => {


Just getting rid of the catch-all to be more rigorous

Jefffrey · 2026-01-24T08:11:35Z

                    _ => unreachable!("Invalid dictionary keys type: {}", key_type),
                }
            }
+            DataType::RunEndEncoded(run_ends_field, value_field) => {


We're building the runarray efficiently here, as unlike dictionary above which would require keeping a hashmap of values to build an efficient dictionary array, run arrays are simpler in that we just need to track when a new run starts.

Most of the verbosity here is related to destructuring input ScalarValues and ensuring we have consistent types from them.

Jefffrey · 2026-01-24T08:12:53Z

+                    let run_ends = PrimitiveArray::<R>::from_iter_values(run_ends);
+                    let values = ScalarValue::iter_to_array(value_scalars)?;
+
+                    // Using ArrayDataBuilder so we can maintain the fields


I think this is the only way to construct runarrays with fields we want, since try_new creates the fields for us:

https://github.com/apache/arrow-rs/blob/ddb6c42194fa45516e1bd4a27cdacf10fda56b5a/arrow-array/src/array/run_array.rs#L99-L105

Jefffrey · 2026-01-24T08:16:01Z

+        );
+        let err = scalar.eq_array(&run_array, 1).unwrap_err();
+        let expected = "Internal error: could not cast array of type Float32 to arrow_array::array::primitive_array::PrimitiveArray<arrow_array::types::Float64Type>";
+        assert!(err.to_string().starts_with(expected));


Needed to use starts_with since backtrace feature can affect the error message, so direct equality can succeed for cargo test but fail in CI

Jefffrey · 2026-01-24T08:17:06Z

-                return Err(Error::General(
-                    "Proto serialization error: The RunEndEncoded data type is not yet supported".to_owned()
-                ))
+            DataType::Decimal32(precision, scale) => {


Only change here is for RunEndEncoded; for some reason other formatting changes were applied for the other arms here

alamb · 2026-01-27T22:45:38Z

FYI @brancz

alamb

I won't say I reviewed every line of this PR carefully but I did read them all and they look structurally good to me -- thank you for pushing this along @Jefffrey

brancz · 2026-01-28T07:33:39Z

            }
            (Dictionary(_, _), _) => None,
+            (RunEndEncoded(rf1, vf1, v1), RunEndEncoded(rf2, vf2, v2)) => {
+                // Don't compare if the run ends fields don't match (it is effectively a different datatype)


I'm not sure this is exactly what we want. The run arrays could be logically identical, but their index types might differ. I don't think we'd want the scalar not to equal in that case. I realize that's not what we have for dictionaries either, but is that really the intention of scalars? My understanding has always been that the integer width of codes should be irrelevant from a logical equality perspective.

I'm not sure if we want that logic as this level; for example if we fix PartialOrd here to compare REE/Dicts based on inner values only, then we'd probably have to do the same for PartialEq right? But then we run into an issue with Hash not being consistent unless we also fix Hash 🤔

I think it might be better to leave these as is, and if we want proper comparison it would make more sense to do at a high level (e.g. via type coercion)

I do believe that to be the right path ultimately, but I can agree to starting with this and consistently change things as a follow up.

Let's file a follow on ticket.

Reconsider equality of ScalarValues for encoded types (REE/Dictionary) #20121

alamb · 2026-02-02T23:49:20Z

Thanks again @Jefffrey and @brancz

vegarsti · 2026-02-03T09:07:21Z

Nice!

## Which issue does this PR close?  - Closes apache#18563 ## Rationale for this change  Support RunEndEncoded scalar values, similar to how we support for Dictionary. ## What changes are included in this PR?  - Add new `ScalarValue::RunEndEncoded` enum variant - Fix `ScalarValue::new_default` to support `Decimal32` and `Decimal64` - Support RunEndEncoded type in proto for both `ScalarValue` message and `ArrowType` message ## Are these changes tested?  Added tests. ## Are there any user-facing changes?  New variant for `ScalarValue` Protobuf changes to support RunEndEncoded type  --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

- Closes apache#18563  Support RunEndEncoded scalar values, similar to how we support for Dictionary.  - Add new `ScalarValue::RunEndEncoded` enum variant - Fix `ScalarValue::new_default` to support `Decimal32` and `Decimal64` - Support RunEndEncoded type in proto for both `ScalarValue` message and `ArrowType` message  Added tests.  New variant for `ScalarValue` Protobuf changes to support RunEndEncoded type  --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Jefffrey added 2 commits January 20, 2026 10:15

Add ScalarValue::RunEndEncoded variant

8c7740c

Store fields in ScalarValue::RunEndEncoded instead of just run-ends…

33f08f0

… type

github-actions bot added sql SQL Planner common Related to common crate proto Related to proto crate labels Jan 20, 2026

Jefffrey changed the title ~~Ree scalarvalue~~ Add ScalarValue::RunEndEncoded variant Jan 20, 2026

Jefffrey added 3 commits January 24, 2026 16:20

Merge branch 'main' into ree-scalarvalue

a13b948

fix error asserts to account for backtrace

af2b46e

fix

f69ee86

Jefffrey commented Jan 24, 2026

View reviewed changes

fix proto field name & comment

1c0b400

Jefffrey marked this pull request as ready for review January 24, 2026 08:38

Merge branch 'main' into ree-scalarvalue

1293f58

alamb added the api change Changes the API exposed to users of the crate label Jan 27, 2026

alamb approved these changes Jan 27, 2026

View reviewed changes

Comment thread datafusion/common/src/scalar/mod.rs

brancz reviewed Jan 28, 2026

View reviewed changes

brancz approved these changes Jan 30, 2026

View reviewed changes

Merge branch 'main' into ree-scalarvalue

759c0b5

alamb added this pull request to the merge queue Feb 2, 2026

Merged via the queue into apache:main with commit 39da29f Feb 2, 2026
28 checks passed

Jefffrey deleted the ree-scalarvalue branch February 3, 2026 02:37

Jefffrey mentioned this pull request Feb 3, 2026

Reconsider equality of ScalarValues for encoded types (REE/Dictionary) #20121

Open

		/// (run-ends field, value field, value)
		RunEndEncoded(FieldRef, FieldRef, Box<ScalarValue>),

Conversation

Jefffrey commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 27, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Feb 2, 2026

Uh oh!

Uh oh!

vegarsti commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jefffrey commented Jan 20, 2026 •

edited

Loading